Mixed reality sensor suite and interface for physical region enhancement

ABSTRACT

Disclosed herein is a mixed reality application to use a multi-channel audio input to identify a character and origin of a given sound, then present a visual representation of the given sound on a near eye display. The visual representation including a vector to the source of the sound. The visual representation further including graphical elements that describe various attributes of the given sound including the magnitude, directionality, source, and threat level. Where the source of the given sound is moving, the visual representation shifts to illustrate the movement.

CROSS-REFERENCE TO RELATED APPLICATION

The present disclosure is a continuation of U.S. patent application Ser.No. 17/201,123, titled MIXED REALITY SENSOR SUITE AND INTERFACE FORPHYSICAL REGION ENHANCEMENT”, filed Mar. 15, 2021, which is acontinuation of U.S. patent application Ser. No. 16/443,568, titled“MIXED REALITY SENSOR SUITE AND INTERFACE FOR PHYSICAL REGIONENHANCEMENT”, filed on Jun. 17, 2019, and further claims the benefit ofpriority under 35 U.S.C. § 119(e) of U.S. Provisional Patent ApplicationSer. No. 62/685,841, titled “MIXED REALITY SENSOR SUITE AND INTERFACEFOR PHYSICAL REGION ENHANCEMENT,” which was filed on Jun. 15, 2018. Theentire contents of the aforementioned patent applications are herebyexplicitly incorporated by reference into the present disclosure for allpurposes.

TECHNICAL FIELD

The present disclosure relates to mixed reality systems in general, andmore particularly, to augmented reality systems that provideenvironmental information to a user.

BACKGROUND

Virtual reality (VR) and augmented reality (AR) environments aregenerated by computer data simulation. Virtual content can immerse auser in a simulated environment through the use of wearing a near to eyedisplay (NED). The NED embodies a transparent display element thatallows virtual content to be displayed for a user to experience throughvisual perception, auditory detection or tactical interaction.

VR and AR systems provide users with entertaining, immersivethree-dimensional (3D) virtual environments in which they can visually(and sometimes audibly) experience things they might not normallyexperience in real life.

SUMMARY

The present disclosure is generally directed to the analysis of audiodata and the presentation of information based on the analysis. Morespecifically, the present disclosure is directed to the analysis ofaudio data to determine a location and/or classification of source(s) ofsound(s) described by the audio data, and the presentation in a displaydevice of graphic element(s) that indicate the location of the source(s)relative to the display device and/or the classification of thesource(s).

Embodiments of the present disclosure include a computer-implementedmethod that performs the following operations: receiving audio datacollected by at least one audio input device, the audio data describingone or more sounds originating from a source in proximity to a displaydevice; analyzing the audio data to determine a location of the sourceof the one or more sounds relative to the display device; andpresenting, through the display device, a graphic element that indicatesthe source of the one or more sounds at the location relative to thedisplay device.

Embodiments of the present disclosure can also optionally include one ormore of the following aspects: the at least one audio input deviceincludes one or more of a microphone and a peripheral sensor device; theaudio data describes at least one sound at a frequency that is outside arange of frequencies audible to humans; analyzing the audio dataincludes filtering the audio data to modulate or remove one or morerepeated sounds; analyzing the audio data includes determining aclassification of the source of the one or more sounds; the graphicelement that is presented through the display device includes theclassification of the source; determining the classification of thesource includes comparing the audio data to a plurality of sounds havingpreviously determined classifications; the classification is based atleast partly on a distance between the source and the display device,the distance determined through the analyzing of the audio data;analyzing the audio data includes determining a direction of movement ofthe source relative to the display device; the graphic element that ispresented through the display device indicates the direction ofmovement; the display device is a near to eye display (NED); and/or theoperations further include transmitting, to at least one other displaydevice, a signal that includes information describing the source,wherein the signal causes each of the at least one other display deviceto present a respective graphic element that indicates the source of theone or more sounds at a respective location relative to the respectiveother display device.

Embodiments of the present disclosure also include a display systemconfigured to perform operations described herein. Embodiments of thepresent disclosure also include memory (e.g., computer-readable storagemedia) that stores instructions that are executable perform operationsdescribed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

One or more embodiments of the present disclosure are illustrated by wayof example and not limitation in the figures of the accompanyingdrawings, in which like references indicate similar elements.

FIG. 1 illustrates an example of a mixed reality system worn by a user.

FIG. 2 illustrates that microphones within a mixed reality system cancommunicate with each other.

FIG. 3 is a flowchart of the data analysis pipeline.

FIG. 4 is a flowchart to determine whether multiple audio inputs arerelated.

FIG. 5 illustrates that two users wearing near-to-eye displays caninteract wirelessly with other users wearing similar devices.

FIG. 6 illustrates a user's view of an augmented reality scene using anexample near-to-eye display (NED).

FIG. 7A illustrates a user's view of an augmented reality scene using anexample NED with vectors.

FIG. 7B illustrates a user's view of an augmented reality scene using anexample NED with vectors after the user has shifted their viewing angle.

FIG. 8 is a block diagram showing the various functional components of anear-to-eye display.

DETAILED DESCRIPTION

In this description, references to “an embodiment,” “one embodiment” orthe like, mean that the particular feature, function, structure orcharacteristic being described is included in at least one embodiment ofthe technique introduced here. Occurrences of such phrases in thisspecification do not necessarily all refer to the same embodiment. Onthe other hand, the embodiments referred to also are not necessarilymutually exclusive.

Virtual reality (VR) and augmented reality (AR) systems disclosed hereincan include a display which presents computer-generated imagery to auser. In some embodiments, the display systems are wearable, which mayadvantageously provide a more immersive VR or AR experience.

The techniques and systems introduced here are meant to provide sensoryinformation regarding a user's environment (e.g., audio) to a userthrough a near-to-eye display (NED). The system can incorporate a suiteof sensors to take in sensory information (e.g., audio) and atransparent display to display the sensory information to the user. Thedevice can also connect to another sensory device, such as the TacticalCommunication and Protective System (TCAPS) to collect similarinformation. The sensory information received from a sensory device caninclude, but is not limited to, multi-channel audio, video in visiblelight or another portion of the electromagnetic spectrum (such asinfrared).

After collecting the sensory information, the device analyzes the inputto compute actionable data from the relevant characteristics of thesensory information. The actionable data may include the probability ofthreat of a given source of sensory data, directionality and/or originof audio, and location of objects of interest. Finally, the sensoryinformation is displayed graphically through a NED.

FIG. 1 illustrates an example of a mixed reality system (MR system) 10worn by user 12. The MR system 10 includes a frame 14, microphone 16,speaker 18, wired or wireless communication 20, various mechanical andelectronic modules and systems (22, 28, and 30) to support thefunctioning of the MR system 10, and display 32. The MR system 10 makesuse of audio input from at least two microphones 16, positioned ateither side of the user's head or body (e.g., over the wearer's ears, ateither shoulder, or another body mounted location that enables sounddirectionality). The display 32 may be physically coupled to a frame 14,which is wearable by user 12 and which is configured to position thedisplay 32 in front of the eyes of the user 12. In some embodiments, aspeaker 18 and microphone 16 are physically coupled to the frame 14 andpositioned adjacent to the ear canal of the user 12. In otherembodiments, the speaker 18 and microphone 16 are co-located and inother embodiments, they are not. In other embodiments, the microphone 16is separate from the frame 14 and supplies input to the NED 11. Thedisplay 32 is operatively coupled, such as by wired or wirelessconnection 20, to a local data processing module 22 which may be mountedin a variety of configurations, such as fixedly attached to the frame14, affixed to a helmet or hat worn by the user 12, or fixedly attachedby a belt-coupling configuration worn by the user 12, or can otherwisebe removably attached to the user 12.

In some embodiments, the remote processing module 28 may include one ormore processors configured to analyze and process data (e.g., sensorydata and/or audio information). In some embodiments, the remote datarepository 30 may comprise a digital data storage facility, which can beavailable through the internet or other network configurations. In otherembodiments, all data is stored and all computations are performed inthe local processing and data module 22, allowing fully autonomous use.In other embodiments, at least some processing and storage can be doneon the remote processing module 28. Network transceivers 24, 26 enablecommunication through the internet or other network configurations tothe remote processing module 28.

In some embodiments, the local processing and data module 22 may includea processor, as well as digital memory, such as non-volatile memory(e.g., flash memory), both of which can be utilized to assist theprocessing and storing of data. The local processing and data module 22may also store data from sensory devices such as Tactical Communicationand Protective System (TCAPS), microphone 16, speaker 18, image capturedevices (e.g., cameras), GPS units and other sensory devices. The localprocessing and data module 22 can be operatively coupled bycommunication links 24, 26, such as by wired or wireless communicationlinks, to the remote processing module 28 and remote data repository 30.Remote modules 28, 30 become a resource to the local processing and datamodule 22 due to this connection via wired or wireless links 24, 26.

In some embodiments, NED 11 can be configured to take input from othersensory devices such as TCAPS. For example, TCAPS can be electronicallycoupled, through a wired or wireless communication, to NED 11 thuseliminating the need for redundant functional elements on NED 11.

In some embodiments, the computer-generated imagery provided via thedisplay 32 can create the impression of being three-dimensional (3D). 3Dimagery can be displayed, for example, by presenting stereoscopicimagery to the user. In some conventional systems, such imagery caninclude separate images of a scene or object from slightly differentperspectives. The separate images can be presented to the user's righteye or left eye, respectively, thus simulating binocular vision andhaving an associated depth perception for each eye.

In some embodiments, a set of one speaker 18 and one microphone 16 arephysically connected to the left ear canal and another set is physicallyconnected to the right ear canal of the user 12, creating two channelsof input. The direction and location of the audio input relative to aposition of a user is determined by a comparison of audio featuresbetween the two-channels of input. Examples of audio features are timelag, wavelength and tone.

Time lag helps determine direction and location because, for example, ifa sound arrives at an angle to the right or left of the user's 12 face,the sound does not reach both ears at the same time. Sound is staggeredbetween ears because the sound waves travel a longer distance beforereaching the left microphone 16. The comparative discrepancy between theaudio input arriving at the left and right microphones 16 enable the MRsystem 10 to determine directionality of the audio input. In someembodiments, the local processing and data module 22 registers the timelag and inform the user 12 of the location of the original sound. Inother embodiments, the remote processing module 28, separately or incombination with the local processing and data module 22, can determinedirectionality and origin of sound.

Wavelength is another factor that helps to determine the sound origin,particularly with treble sounds. Sounds can be separated into highbandwidth and low bandwidth frequencies. Treble sounds are indicative ofhigh bandwidth frequencies and thus can be differentiated from low orbase bandwidth noises. In such situations, the head of a user 12functions as a filter and prevents sounds from traveling around the headto the opposite ear. In some embodiments, local processing and datamodule 22 can register and process the variation in sound between thetwo-channels of input. In other embodiments, the remote processingmodule 28, separately or in combination with local processing and datamodule 22, can register and process the difference in sounds between thetwo-channels of input.

Tone can help determine the location of sound in situations where thereis no time lag between the ears (e.g., when the sound comes from above,below or symmetrically in front of the face). Tone can be recognized byvibration regularity. For example, a simple tone has only one frequency,with varying amplitude. A complex tone, conversely, can have multiplefrequencies. In some embodiments, local processing and data module 22can register and process the tone. In other embodiments, the remoteprocessing module 28, separately or in combination with local processingand data module 22, can register and process the tone.

FIG. 2 depicts three microphones 16 which are wired or wirelesslyconnected by link 38. Microphones 16 can be operatively coupled to MRsystem 10, physically coupled to the frame 14, affixed to the user's 12body, electronically coupled to a speaker 18 or coupled to other areaswhich can help determine directionality of a sound. In some embodiments,a set of multiple microphones 16A-C are included on each side of theuser 12's head/body. For example, the set could include threemicrophones per side of a user which are positioned in order todetermine the relative height and direction of the audio input based ona single side. The distance between each microphone 16A-C is known;thus, received audio input is compared (e.g., time lag, wavelength andtone) amongst each of the microphones 16A-C in order to calculatewhether the sound comes from the front or behind, and above or below.Each microphone receives a different sound than other microphones in theset. Processing modules 22, 28 are thus able to accurately determine thedirection and location of a sound. The use of more microphones increasesthe accuracy of the system.

In some embodiments, microphone 16 can be implemented mechanically suchas by an acoustic resonator (e.g., tuning rod). An acoustic resonatorvibrates in the same frequency as received audio signals. Vibration fromthe acoustic resonator can then be processed as audio input by one orboth of local processing and data module 22 or remote processing module28.

FIG. 3 shows the general flow diagram of a data processing pipeline 300.In step 302, the user activates the MR system 10. The user can activatethe MR system 10 by using a button, switch, or other operative controlelement.

In step 304, the microphones 16 receive an audio input. In typicalcases, the audio input from each microphone includes wavelength andamplitude. In some embodiments, the audio input could be received from aperipheral sensory device such as TCAPS. The audio input can alsoinclude audio that is outside the range of frequencies that is audibleto humans.

Additionally, the MR system 10 is capable of sound modulation to filterout repeated sound waves. Since a user 12 may receive sound waves fromall directions, the user likely receives both direct and reflected soundwaves originating from the same source. However, the reflected soundwaves can be modulated or removed from the audio data. Modulation orremoval of the repeated sound waves can be done by analyzing the audiofeatures to recognize the time lag between the original sound and therepeated sound. Thereby, the processing modules 22, 28 can remove thereflected sound.

In step 306, the processing modules 22, 28 determines the originationlocation and magnitude of the audio input. To do so, the processingmodules 22, 28 can use pre-loaded machine learning models to analyze theaudio input. For example, the processing modules 22, 28 can use input(e.g., wavelength, tone, and time lag between microphones) from multiplemicrophones to determine the origin of a noise. Additionally, otheraudio characteristics can be determined such as whether a sound is areflection (e.g., echo).

Audio characteristics are used to compare the audio input to thepre-loaded machine learning model/neural network that includes aplurality of known sounds and the various ways in which those soundscould manifest. The more complete the model, the more sounds can becharacterized. The model can identify multiple sounds occurring within asingle audio input in order to aid in classifying the audio input.

In some cases, the origin of audio input is a moving object/entity. Acontinuous audio input can be interpreted to determine whether theorigin of the audio input is moving towards or away from the user 12.Determination of whether an audio input is moving towards or away fromthe user 12 is done by processing modules 22, 28 and recognizing theaudio input's Doppler characteristics (e.g., frequency and wavelength).When a sound is moving closer to a user 12 or the user 12 is movingcloser to a sound, each successive wave crest is emitted from a positioncloser to the user 12 than the previous wave. Therefore, each successivewave is received by a set of microphones 16 in slightly less time thanthe previous wave. Since the time is shorter, the frequency increases.Conversely, when a sound is moving away, each successive wave crestarrives in a slightly longer time period than the previous wave, thusdecreasing the frequency. Therefore, the processing module 22, 28 caninterpret a variation in arrival time to indicate whether a sound ismoving farther away. A graphic representation of whether a sound ismoving towards or away from a user 12 can include an arrow pointing tothe user 12 or away from user 12 to indicate whether the sound is travelto or away from the user 12.

In another embodiment, processing modules 22, 28 can use the principlesof redshift and blueshift to deduce whether a sound is traveling to oraway from user 12. When a sound travels away from the user, redshiftoccurs because the wavelength of the electromagnetic radiation (e.g.,light) of the source increases. The wavelength increases because eachsuccessive wave crest takes a slightly longer time to arrive at MRsystem 10 than the previous wave crest. Conversely, when a sound movestowards the user 12, blueshift occurs because the wavelength of theelectromagnetic radiation (e.g., light) of the source decreases. Thewavelength decreases because each successive crest takes a slightlyshorter time period to arrive at MR system 10 than the previous crest.The MR system 10 can detect changes in electromagnetic radiation byusing video data to assess the wavelength.

In step 308, audio input is classified. In some embodiments, audio inputcan be classified into one or more of categories such as who or whatmade the noise, by direction of origin, by decibels, by relation to thetask being done by the user and by importance to other MR system users.For example, in a war zone, multiple MR system users may be approachingenemy territory together; however, each user may have a different lineof sight or be closer to certain sounds. Therefore, noises from guns maybe classified as important to the local MR user 12 and to other MRsystem users as well. A noise from a gun to the left of a user can beclassified as information which is to be passed onto nearby MR systemsas noise which originated to the left of the user. By passinginformation of the gun noise (e.g., location, directionality,classification) to other MR system users, they become aware of thesituation quickly and can take action accordingly (e.g., take cover,retaliate). Processing modules 22, 28 classify based on, the proximityto the user 12, the wavelength, the frequency, the tone, theenvironment, and other related characteristics.

In step 310, the processing modules 22, 28 evaluate the threat level ofthe audio input. The threat level can be based on how the sound wasclassified using the learned model. For example, audio input can beclassified as highly dangerous based on the sound of gunfire or notdangerous based on the sound of children playing. Additionally, thethreat level can be based on input from MR systems nearby, the proximityof the sound, or activity performance of the user. In some embodiments,threat level can be a separate category for a sound to be classifiedunder.

In step 312 and 314, the data receives a corresponding graphicalrepresentation which can be displayed on display 32. For example, thesound of gun fire from the left causes generation of a graphic of anarrow pointing left and a corresponding threat level graphic (e.g., agun). In other embodiments, the graphical representation depictingorigination, direction and threat level is the same. For example, thesound of gun fire from the left can be depicted by a red vector pointingleft, indicating the danger, direction and origin.

To perform the steps of pipeline 300, machine learning models can bebased on, for example, one or a combination of linear or logisticregression models, linear discriminant analysis, classification andregression trees, Naïve Bayes algorithms, K-nearest neighbors (KNN)algorithms, learning vector quantization models, bagging and randomforest models, boosting and adaboost methods or other related models oralgorithms.

In some embodiments, audio and threat level data are classified based onpredetermined categories. For example, audio can be classified in arange of danger levels (e.g., from high to low) based on the proximityand type of noise. Proximity of the noise can be determined by usingtwo-channels of input and either the local processing and data module 22or the remote processing module 28 to assess the audio features (e.g.,time lag, wavelength, and tone). The type of noise can be determined byan iterative process of comparing the noise with noises in a database ofpreloaded sounds to find the best match.

FIG. 4 shows the general flow diagram 400 of how the MR system 10 usesthe relatability of audio inputs to evaluate the threat level of thesame. In step 402, MR system 10 is initialized. In step 404, multipleaudio inputs are received by the system. The multiple audio inputs maybe received in quick succession (0-2 seconds), or over a longer period(>2 seconds). In step 406, the relatability of the sounds is determinedusing machine learning models. Factors such as direction, time betweensounds, audio characteristics of the sounds, and relative location ofthe multiple sounds can help determine their relatability. The source ofmany sounds is often connected (e.g., a cause and an effect). Forexample, the sound of a bullet being fired, and the sound of a bulletimpacting a target (or missing the target) are related. Another exampleis the sound of a car starting and then the sound of a car moving. Oncerelatability of the sounds is determined, in step 408, relatability canbecome a factor in evaluating the threat level of said sounds. Returningto the example of gun fire, a sound of a gun firing and a correspondingsound of a bullet hitting the ground can be considered to be inrelation. If the bullet impacts close to the user 12, the threat levelis obviously significantly higher than if the bullet arrives far awayfrom the user 12. Another example can be if there are successive soundsof guns firing, which can be seen as a high threat level.

In step 410, the device displays the graphical representation of thethreat level of the audio input. The graphical representation can vary(e.g., order and location of display of varied audio inputs) based on,for example, user preferences or machine learning models. In someembodiments, the user 12 can elect preferences such as to give priorityto certain information (e.g., direction or threat level). An option toelect preferences may be presented to the user 12 before initializingthe MR system 10. The preferences may include an option to give priorityto certain classifications over others. For example, a user in amilitary zone may want the indication of gun fire to take up 30% of thescreen, whereas less threatening indications use 10% of the screen. Inother situations, the user 12 may need to encircle an enemy withincrosshairs, thus crosshairs may take 50% of the screen.

In other embodiments, machine learning models can prioritize importancebased on, for example, the user 12's environment, past preferences, ortype of audio. Moreover, the MR system 10, can deduce the user 12'slocation from input from a GPS and thus change the display to showpertinent information. For example, if the user 12 is in a dense forest,sounds of predators may be of high importance. In other embodiments, theuser 12's past preferences may be an indication of how to adapt thedisplay. For example, if the user 12 usually prioritizes threat levelwhen using the MR system 10 in the dark, then the machine learningmodels learn to prioritize display of threat level as a pre-setpreference.

Additionally, the graphical representation can vary based on differentkinds of sounds. The graphical representation can vary based on userpertinence. For example, the location of where a bullet impacted theground is not as significant as to the location of the gunman.Therefore, the MR system 10 may only display a single vector showing thelocation of the gun. In other embodiments, the MR system 10 can showmultiple vectors which point to related sounds. In the same example asthe gun firing a bullet, the MR system 10 may display two vectors. Onethat points to the gun origin and another that points to the location ofimpact. Both these vectors can be similarly colored to indicate therelation of the event and sound.

In some embodiments, NED 11 can incorporate other sensory devices suchas TCAPS, a camera (e.g., thermal camera, depth camera), heart ratesensor, GPS, accuracy assist, rapid target acquisition, and otherdevices which can be added to help a user in particular situations. Forexample, a soldier may benefit from an accuracy assist camera thatquickly focuses on a target. Thus, NED 11 can incorporate a movinggraphic that encircles an enemy (e.g., crosshair). A soldier may alsobenefit from a heart rate sensor to alert oneself when a break isnecessary. NED 11 is able to display a heart pulse graphic to indicatethe user's 12 corresponding heart rate.

Additionally, different sensory devices may be needed based on the rankof a solider. For example, a foot solider may need to maximize sensoryassistance devices to be aware of the surroundings. A sniper, on theother hand, may need auditory assistance to locate a target whilelowering the noise produced by firing a gun. A general, may needcommunication assistance to be able to talk to all the soldiers andassistance in guiding soldiers in a strategic manner. User-specificneeds can be achieved by, for example, prioritizing the display to showonly what is crucial to the specific user. In another example, the MRsystem 10 could have a pre-set user interface. Thus, the user 12 needonly select which profile is preferred before operating the MR system10.

FIG. 5 shows several MR system users 12 wirelessly communicating overinterface 42 with each other. The interface 42 can be the internet orother network configurations and wired or wireless. The data sharedbetween the MR system users 12 can be audio data, proximity data, threatlevel data, and other related data. For example, a gunshot heard in thevicinity of all MR system users 12 can be shared with other MR systemusers 12 to best determine the location, direction and threat level ofthe noise. In some embodiments, the local processing and data module 22can process the data collected from the local NED 11 and subsequentlypass said data through a communication interface 42 to other NEDs 11.

In another embodiment, the processing modules 22, 28 can remove, ignore,or dampen unthreatening sounds based on information from other MRsystems 10. For example, sounds created by friendly MR system users 12can be removed, ignored or dampened if they don't pose a threat, such asfootsteps. Removing, ignoring or dampening a non-threatening sound mayallow the user 12 to focus on the sounds which matter (e.g., the soundswhich pose a threat). Additionally, the processing modules 22, 28 canremove, ignore or dampen certain sounds based on information from otherMR systems 10 by assessing the other MR system's 10 location,recognition and/or classification of similar sounds.

In other embodiments, MR system 10 can deduce whether a sound is notfrom the original source (e.g., bounced off a wall such as an echo). Todeduce whether a sound is not from the original source, the MR system10, for example, can get input from other MR system 10 users 12 such asvideo data, or origination data. For example, if one MR system 10received audio with certain audio features, while another received asimilar sound (e.g., similar in frequency and amplitude) but withdifferent audio features, then the sound can be flagged for furtheranalysis. MR system 10 can use audio features of echoes, such as a timedelay from the original sound, for analysis. In echoes, the time delayis proportional to the distance to the reflecting surface from thesource of the sound and the listener. Thus, a determination of whetherthe sound was an echo can be done by comparing the audio features of thesound from each MR system. Moreover, the origination location can alsobe determined by geometric principles (e.g., Pythagorean theorem).

In other embodiments, video data can be used to find the originationlocation of a sound. For example, if the source of the sound is pointingto a wall, the processing modules 22, 28 can flag a reflected sound asan echo. Next, video data can be used to find the angle of the wall,which can then be used to calculate the original source's location bygeometric principles (e.g., Pythagorean theorem).

In other embodiments, previous origination data can be used topredetermine an outcome or the source of the sound. For example, if thesource of a sound has been identified as a moving object, when theobject moves out of eye-sight but makes a sound, the MR system 10 canidentify the source based on the trajectory and speed of the source. Inother embodiments, the source may be stationary but the user 12 has theability to move where the source isn't visible. However, before the user12 moves, the MR system 10 is able to register audio from the source.Once the user 12 has moved out of sight and the source makes a similarnoise, the previous audio data can be matched (e.g., by wavelength,tone, time lag) with the current audio data to indicate the likelysource of the sound.

In another embodiment, sound propagation maps (e.g., sonar maps) can beused to find the original source of a sound. There are two types ofsound propagation techniques: passive and active. Under passivepropagation, the MR system 10 only listens for sounds. Under activepropagation, the MR system 10 emits pulses of sounds and listens forechoes, thus can locate objects and formulate their general shape. Forexample, a MR system 10 can receive a sound and through active soundpropagation can detect if the sound came from a wall. Furthermore, theMR system 10 can calculate the angle of the wall and derive the locationof the original source using geometric principles (e.g., Pythagoreantheorem).

FIG. 6 illustrates an example of a user's view of an augmented realityscene using an example NED 11. NED 11 incorporates a real-world scene 46with augmented reality graphical representations 44, 45, 46, 48, 50, 52,and 53. Each augmented reality graphic can indicate the level of threat,direction of sound or what produced the sound. For example, the soundbar 44 indicates the level of sound. The danger symbol 48 shows whetherthe sound is dangerous or not through a variation in color. The arrow 50shows the direction of the sound. Symbol 52 shows what produced thesound (e.g., in this case a human). Crosshair 45 can assist inencircling a target. Heart symbol 53 can show the user's 12 heart rateby pulsing.

In another embodiment, NED 11 can use different graphics and symbols.For example, if a user of NED 11 is in the middle of a war zone, symbol52 can be a gun or grenade. Danger symbol 48 can indicate whether thecurrent direction is safe or unsafe based on data collected fromcommunicating with other MR system users 12. The arrow 50 can be avector that indicates magnitude and direction of the sound. In anotherembodiment, the graphical representations can be static, while the colorchanges. For example, the danger symbol 48 can flash red when the user12 is approaching a dangerous area and then flash green when the user isapproaching a safe zone.

FIG. 7A shows another embodiment of the NED 11. The audio inputs 54 areidentified by vectors 56. Vectors 56 locate the source of the sound andcan vary in thickness, color and or type (e.g., solid, dashed orflashing). The variations can be modified by, for example, the user 12,set by machine learning models or based on the environment. In oneembodiment, the thickness may be an indication of decibels, the colormay be indication of threat level, and the type may be an indication ofthe relation to another sound with a similar type of vector. In anotherembodiment, the vector may change colors based on the environment toensure there is enough contrast to recognize the AR graphics. Forexample, if the user 12 is in a forest, a green graphic may not haveenough contrast. In such a situation, the MR system 10 can use GPS andvideo information to analyze the environment and use other colors toincrease the contrast.

FIG. 7B shows that vectors 56 can maintain directionality (e.g., pointto the source of the sound) even while the user 12 turns their head. InFIG. 7B, the user 12 has shifted their view to the right. However,vectors 56 continue to the pointed to the audio inputs 54 in real-time.To maintain directionality, the processing modules 22, 28 can analyze acertain sound for audio features (e.g., wavelength, tone, time lag,amplitude). By analyzing audio features, the location can be deduced asnon-stationary, which prompts the display system to adjust the vectoraccordingly. In another embodiment, when the user 12 may be stationarybut the audio input 54 may move (e.g., an airplane), the vector 56 cancontinue to update in real-time due to differences in audio features andpoint to the moving audio input 54. For example, the sound of anairplane moving away from the user has a different frequency with eachcrest due to the soundwaves arriving slightly later with each crest.Therefore, the processing modules 22, 28 can deduce the direction ofmovement and adjust the displayed vector accordingly.

In another embodiment, vectors 56 can indicate threat level or noiselevel by appearing thicker or narrower, transitioning from a solid to adotted type, flashing and/or changing colors. Threat levels can be basedon proximity, classification, relation to other noises and informationfrom other MR users 12 (e.g., FIG. 3 and FIG. 4.). Once classified, thegraphical representation can vary based on, for example, user 12'spreference, machine learning models and the environment. For example, ifthe user 12 is in the middle of a war zone, the threat level vector 56may appear to flash in a bright color to garner the full attention ofthe user 12. In another embodiment, vectors 56 can be incorporated withother graphical representations such as those shown in FIG. 6.

In another embodiment, the display can be optimized to better assist theuser 12 to operate efficiently and process information quickly. In somesituations, a user 12 may benefit from having only one graphicdisplayed. For example, when the user 12 wants to encircle an enemy incrosshairs, only the crosshair graphic can be displayed. In othersituations, more graphics may be needed because the user 12 wants asmuch information as possible. For example, if the user 12 is lost in themountains, gathering information could be crucial to finding a way backto safety. In other embodiments, there can also be nothing displayedbecause the user 12 needs to focus on the real-world.

The MR system 10 is configurable and adaptable based on, the user 12'spreferences, machine learning models or the environment. A configurableand adaptable MR system 10 allows for the user 12 to get crucialinformation quickly, process information quickly, and to be efficient instressful situations.

FIG. 8 depicts a block diagram of various functional components of an MRsystem 10, according to some embodiments. The functional components ofan MR system 10, in FIG. 8, includes one or more instance of each of thefollowing: processor 58, main memory 60, drive unit 62, static memory66, output interface 68, illumination module/EM emitter 70 and depthcamera 72, all electronically coupled together by an interconnect, BUS76. In some embodiments, the interconnect 76 can be one or more of wiredor wireless connectors such as adapter, traces and other conventionalconnectors.

The drive unit 62 includes a machine-readable medium 64 in which a setof executable instruction is stored i.e., software instructions 74,embodying any one, or all, of the methodologies described herein. Thesoftware instructions 74 are also shown to reside, completely or atleast partially, within the main memory 60 and/or within the processor58. The software instructions 74 may further be transmitted or receivedover a network by means of a network interface device 68.

In contrast to the system 78 discussed above, a different embodimentuses logic circuitry instead of computer-executed instructions toimplement processing entities. Depending upon the particularrequirements of the application in the areas of speed, expense, toolingcosts, and the like, this logic may be implemented by constructing anapplication-specific integrated circuit (ASIC) having thousands of tinyintegrated transistors. Such an ASIC may be implemented with CMOS(complementary metal oxide semiconductor), TTL (transistor-transistorlogic), VLSI (very large systems integration), or another suitableconstruction. Other alternatives include a digital signal processingchip (DSP), discrete circuitry (such as resistors, capacitors, diodes,inductors, and transistors), field programmable gate array (FPGA),programmable logic array (PLA), programmable logic device (PLD), and thelike.

Software or firmware to implement the embodiments introduced here may bestored on a machine-readable storage medium and may be executed by oneor more general-purpose or special-purpose programmable microprocessors.A “machine-readable medium,” as the term is used herein, includes anymechanism that can store information in a form accessible by a machine(a machine may be, for example, a computer, network device, cellularphone, personal digital assistant (PDA), manufacturing tool, any devicewith one or more processors, etc.). For example, a machine-accessiblemedium includes recordable/non-recordable media (e.g., read-only memory(ROM); random access memory (RAM); magnetic disk storage media; opticalstorage media; flash memory devices; etc.), etc.

It is to be understood that embodiments may be used as or to supportsoftware programs or software modules executed upon some form ofprocessing core (such as the CPU of a computer) or otherwise implementedor realized upon or within a system or computer readable medium. Amachine-readable medium includes any mechanism for storing ortransmitting information in a form readable by a machine, e.g., acomputer. For example, a machine-readable medium includes read-onlymemory (ROM); random access memory (RAM); magnetic disk storage media;optical storage media; flash memory devices; electrical, optical,acoustical or other form of propagated signals, for example, carrierwaves, infrared signals, digital signals, etc.; or any other type ofmedia suitable for storing or transmitting information.

Further, it is to be understood that embodiments may include performingoperations and using storage with cloud computing. For the purposes ofdiscussion herein, cloud computing may mean executing algorithms on anynetwork that is accessible by Internet-enabled or network-enableddevices, servers, or clients and that do not require complex hardwareconfigurations, e.g., requiring cables and complex softwareconfigurations, e.g., requiring a consultant to install. For example,embodiments may provide one or more cloud computing solutions thatenable users, e.g., users on the go, to access real-time video deliveryon such Internet-enabled or other network-enabled devices, servers, orclients in accordance with embodiments herein. It further should beappreciated that one or more cloud computing embodiments includereal-time video delivery using mobile devices, tablets, and the like, assuch devices are becoming standard consumer devices.

From the foregoing, it will be appreciated that specific embodiments ofthe invention have been described herein for purposes of illustration,but that various modifications may be made without deviating from thescope of the invention. Accordingly, the invention is not limited exceptas by the appended claims.

1. A method, comprising: receiving respective first and second sounds insuccession; determining whether the first and second sounds are relatedto each other; evaluating a threat level of the first and second soundsbased on a determination of whether the first and second sounds arerelated to each other; and generating a pictogram of the threat level ofthe first and second sounds.
 2. The method of claim 1, whereindetermining whether the first and second sounds are related to eachother comprises analyzing a factor selected from the group consisting ofdirection of the first and second sounds, time between the first andsecond sounds, audio characteristics of the first and second sounds, andrelative location of the first and second sounds.
 3. The method of claim1, wherein determining whether the first and second sounds are relatedto each other comprises: determining that the first sound corresponds toa cause; and determining that the second sound corresponds to an effectof the cause.
 4. The method of claim 3, wherein: the first sound is asound of a bullet being fired, and the second sound is a sound of abullet impacting a target.
 5. The method of claim 3, wherein: the firstsound is a sound of a bullet being fired, and the second sound is asound of a bullet missing a target.
 6. The method of claim 3, wherein:the first sound is a sound of a car starting, and the second sound is asound of a car moving.
 7. The method of claim 3, wherein: the firstsound is a sound of a bullet being fired, and the second sound is asound of a bullet hitting the ground.
 8. The method of claim 7, whereinevaluating a threat level of the first and second sounds comprisesdetermining a distance of the sound of the bullet hitting the groundfrom the user.
 9. The method of claim 1, wherein the first sound is asound of a first bullet being fired, wherein the second sound is a soundof a second bullet being fired wherein evaluating a threat level of thefirst and second sounds comprises determining that first and secondsounds are sounds of respective first and second bullets being fired.10. The method of claim 1, further comprising determining a size of thepictogram based on the threat level of the first and second sounds. 11.A display system, comprising: a display device; at least one processoroperatively coupled to the display device; and memory communicativelycoupled to the at least one processor, the memory storing thereupon aset of instructions which, when executed by the at least one processor,cause the at least one processor to perform a set of acts, the set ofacts comprising: receiving respective first and second sounds insuccession; determining whether the first and second sounds are relatedto each other; evaluating a threat level of the first and second soundsbased on a determination of whether the first and second sounds arerelated to each other; and generating a pictogram of the threat level ofthe first and second sounds.
 12. The display system of claim 11, whereindetermining whether the first and second sounds are related to eachother comprises analyzing a factor selected from the group consisting ofdirection of the first and second sounds, time between the first andsecond sounds, audio characteristics of the first and second sounds, andrelative location of the first and second sounds.
 13. The display systemof claim 11, wherein determining whether the first and second sounds arerelated to each other comprises: determining that the first soundcorresponds to a cause; and determining that the second soundcorresponds to an effect of the cause.
 14. The display system of claim13, wherein: the first sound is a sound of a bullet being fired, and thesecond sound is a sound of a bullet impacting a target.
 15. The displaysystem of claim 13, wherein: the first sound is a sound of a bulletbeing fired, and the second sound is a sound of a bullet missing atarget.
 16. The display system of claim 13, wherein: the first sound isa sound of a car starting, and the second sound is a sound of a carmoving.
 17. The display system of claim 13, wherein: the first sound isa sound of a bullet being fired, and the second sound is a sound of abullet hitting the ground.
 18. The display system of claim 17, whereinevaluating a threat level of the first and second sounds comprisesdetermining a distance of the sound of the bullet hitting the groundfrom the user.
 19. The display system of claim 11, wherein the firstsound is a sound of a first bullet being fired, wherein the second soundis a sound of a second bullet being fired wherein evaluating a threatlevel of the first and second sounds comprises determining that firstand second sounds are sounds of respective first and second bulletsbeing fired.
 20. The display system of claim 11, further comprisingdetermining a size of the pictogram based on the threat level of thefirst and second sounds.